Load packages:
We used the rtweet package to pull Twitter data from the PAC-12 universities. Specifically, we used the university’s admissions Twitter handle if there was one, or the main Twitter handle for the university if there wasn’t one:
# p12 <- c("uaadmissions", "FutureSunDevils", "caladmissions", "UCLAAdmission",
# "futurebuffs", "uoregon", "BeaverVIP", "USCAdmission",
# "engagestanford", "UtahAdmissions", "UW", "WSUPullman")
# p12_full_df <- search_tweets(paste0("from:", p12, collapse = " OR "), n = 500)
#
# saveRDS(p12_full_df, "p12_dataset.RDS")
# Load previously pulled Twitter data
p12_url <- "https://github.com/anyone-can-cook/rclass2/raw/main/data/recruiting/p12_dataset.RDS"
p12_full_df <- readRDS(url(p12_url, "rb"))
# Use subset of data
p12_df <- p12_full_df %>% select("user_id", "created_at", "screen_name", "text", "location")
head(p12_df)
#> # A tibble: 6 x 5
#> user_id created_at screen_name text location
#> <chr> <dttm> <chr> <chr> <chr>
#> 1 22080148 2020-04-25 22:37:18 WSUPullman "Big Dez is headed to I~ Pullman, Wa~
#> 2 22080148 2020-04-23 21:11:49 WSUPullman "Cougar Cheese. That's ~ Pullman, Wa~
#> 3 22080148 2020-04-21 04:00:00 WSUPullman "Darien McLaughlin '19,~ Pullman, Wa~
#> 4 22080148 2020-04-24 03:00:00 WSUPullman "6 houses, one pick. Co~ Pullman, Wa~
#> 5 22080148 2020-04-20 19:00:21 WSUPullman "Why did you choose to ~ Pullman, Wa~
#> 6 22080148 2020-04-20 02:20:01 WSUPullman "Tell us one of your Br~ Pullman, Wa~Here are two useful “cheat sheets” about working with strings and regular expressions
Print these cheat sheets. Make one of them your “go to” cheat sheet
Credit: Regex Humor (Rex Egg)
In rclass1, we introduced strings and some basic functions for working with springs
In rclass2, this “Strings and Regular Expressions” lecture provides deeper knowledge about strings, string functions, and – most importantly – regular expressions.
What are regular expressions? (Geeks for Geeks)
Regular expressions are an efficient way to match different patterns in strings, similar to the ctrl+f or cmd+f function you use to find text in a pdf or word document
For example, regex can be used to match all cases of the exact text "out-of-state". But what makes it so powerful is that we could also have it match different variations or patterns, like "Out-of-state", "out of state", etc.
Credit: Crystal Han, Ozan Jaquette, & Karina Salazar (Recruiting the Out-Of-State University)
In her popular STAT545 class Jenny Bryan, professor of statistics at University of British Columbia, describes regular expressions (regex) as:
A God-awful and powerful language for expressing patterns to match in text or for search-and-replace. Frequently described as “write only”, because regular expressions are easier to write than to read/understand. And they are not particularly easy to write."
Yes, learning regular expressions is painful. So why are we making you do this? Because regular expressions are a fundamental building block of data science.
A thing people say is that data science is about trying to find the “signal in the noise”
Another thing people say is that “data science is 80% data cleaning and 20% analysis.”
Much handcrafted work — what data scientists call “data wrangling,” and “data munging” — is still required. Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets.
“Data wrangling is a huge — and surprisingly so — part of the job,” said Monica Rogati, vice president for data science at Jawbone, whose sensor-filled wristband and software track activity, sleep and food consumption, and suggest dietary and health tips based on the numbers. “It’s something that is not appreciated by data civilians. At times, it feels like everything we do.”
“It’s an absolute myth that you can send an algorithm over raw data and have insights pop up,” said Jeffrey Heer, a professor of computer science at the University of Washington and a co-founder of Trifacta, a start-up based in San Francisco.
But if the value [of data science] comes from combining different data sets, so does the headache. Data from sensors, documents, the web and conventional databases all come in different formats. Before a software algorithm can go looking for answers, the data must be cleaned up and converted into a unified form that the algorithm can understand.
So why learn regular expressions? Because regular expressions are THE preeminent tool for identifying data patterns, and cleaning/transforming “noisy” data
This section introduces some prerequisite functions and concepts that will help us learn regular expressions.
str_view() and str_view_all()We introduce the str_view() & str_view_all() functions from the stringr package (part of tidyverse) to help us visualize what is being matched with our regular expressions
The str_view() & str_view_all() functions:
?str_view
?str_view_all
# SYNTAX AND DEFAULT VALUES
str_view(string, pattern, match = NA)
str_view_all(string, pattern, match = NA)str_view() shows the first match of a regex patternstr_view_all() shows all the matches of a regex patternstring: Input vector. Either a character vector, or something coercible to one.pattern: Pattern to look for.
stringi::stringi-search-regex. Control options with regex().match: If TRUE, shows only strings that match the pattern. If FALSE, shows only the strings that don’t match the pattern. Otherwise (the default, NA) displays both matches and non-matches.
Example: Using str_view() & str_view_all() to match literal text
We’ll match text from this string vector:
#p12_df$text[119]
writeLines(p12_df$text[119])
#> "I stand with my colleagues at @UW and America's leading research universities as they take fight to Covid-19 in our labs and hospitals."
#>
#> #ProudToBeOnTheirTeam x #AlwaysCompete x #GoHuskies https://t.co/4YSf4SpPe0
p12_df$text[119] %>% length() # our string has length==1 (i.e., it is a one-element character vector)
#> [1] 1Let’s use these functions to match the exact string "Co" from one of the tweets in our p12_df dataframe. str_view() will show us the first pattern match. Notice that the pattern is case-sensitive, as the "co" in "colleagues" was not matched:
We can use str_view_all() to show all matches, not just the first match:
We can also apply str_view() and str_view_all() to vectors with more than one element
p12_df$text[119] %>% length() # one element
#> [1] 1
p12_df$text %>% length() # many elements
#> [1] 328when applying str_view() to a character vector with more then one element, str_view() shows us the first pattern match for each element (output omitted)
when applying str_view_all() to a character vector with more then one element, str_view_a;;() shows all pattern matches for each element (output omitted)
\) escapeThe concepts “special characters” and “escape sequence” are essential for a deeper understanding of strings and for working with regular expressions. But these concepts are tricky to get your head around, in part because you cannot understand one concept without understanding the other.
Special characters
The literal definition of special characters are characters that are not alphanumeric characters (e.g., \,?, ()
But usually, when the programming world talks about special characters in relation to working with strings, special characters are defined as:
For example, here are two common special characters
\n represents a new line (sometimes called “carriage return”)\t represents a tabThese characters followed by a backslash \ take on a new meaning. The n by itself is just an n. When you add a backslash to the \n you are “escaping it” and making it a special character where \n now represents a newline.
x <- "Hi!\nMy name is\nWhat?\nMy name is\nWho?\nMy name is\nChika-chika\nSlim Shady"
#print(x) # print
writeLines(x)
#> Hi!
#> My name is
#> What?
#> My name is
#> Who?
#> My name is
#> Chika-chika
#> Slim Shady
#Hi!\nMy name is\nWhat?\nMy name is\nWho?\nMy name is\nChika-chika\nSlim ShadyEscape sequences
Definition of escape sequence
\ is called an escape sequence and allows us to include special characters in our strings.”This [Wikipedia quote] about the C programing language is also true for R and most other programming languages:
In C, all escape sequences consist of two or more characters, the first of which is the backslash,
\(called the “Escape character”); the remaining characters determine the interpretation of the escape sequence. For example,\nis an escape sequence that denotes a newline character
Usually, we use the backslash (\) escape character for one of two broad purposes:
\n in our string to insert a newline/carriage return in our string)Using escape backslash escape cahracter \ to enable our string to include a literal character that would otherwise be interpreted by the programming language as a special character
Example: what if we wanted to include quote characters (e.g., single quote ', or double quote ") in our string
If we enclose our string using single quotation marks ', we cannot insert a single quotation mark within the string (code not run)
Similarly, if we enclose our string using double quotation marks ", we cannot insert a double quotation mark within the string (code not run)
Solution, without using backslash (\) escape character
' in our string then enclose the entire string using double quotes "x <- "I am trying to include a single quote ' within my string"
writeLines(x)
#> I am trying to include a single quote ' within my string" in our string then enclose the entire string using double quotes 'x <- 'I am trying to include a double quote " within my string'
writeLines(x)
#> I am trying to include a double quote " within my stringSolution, using backslash (\) escape character
' within our string we can use \' to “escape” a single quotation markmy_string <- 'Escaping single quote \' within single quotes'
writeLines(my_string)
#> Escaping single quote ' within single quotesmy_string <- "Escaping a double quote \" within double quotes"
writeLines(my_string)
#> Escaping a double quote " within double quotes
Similarly, to include a literal backslash \ in the string, we need to escape the backslash with another backslash:
my_string <- "The executable is located in C:\\Program Files\\Git\\bin"
my_string
#> [1] "The executable is located in C:\\Program Files\\Git\\bin"
writeLines(my_string)
#> The executable is located in C:\Program Files\Git\binUsing escape backslash escape cahracter \ to include a special character in our string
We create different versions of an object named my_string that contains special character \t for tab and special character \n for new line
my_string <- "A\tB" # contains \t tab
#my_string
writeLines(my_string)
#> A B
my_string <- "B\nC" # contains \n newline
writeLines(my_string)
#> B
#> C
my_string <- "A\tB\nC\tD" # contains \tab and \n newline
writeLines(my_string)
#> A B
#> C D
Summary: We can use the backslash (\) escape character for:
\n: newline\t: tab\': include literal single quote\": include literal double quote\\: include literal backslashLet’s examine the object my_string (created below) which contains special characters \t for tab and \n for newline
When we print my_string() using the print() function, the output looks different than printing it using the writeLines() function
print() functionprint(my_string)
#> [1] "A\tB\nC\tD"
my_string # same as printing my_string object using print()
#> [1] "A\tB\nC\tD"writeLines() function[CRYSTAL - CAN YOU REFINE/MODIFY THIS TEXT AS YOU SEE FIT] Let’s differentiate (A) string text as it is stored within R from (B) string text as it would be read once special characters (e.g., \n) are compiled
DEFINE WHAT WE MEAN BY “INCLUDE A LITERAL X” VS. WHAT
We can use writeLines() to see the escaped string:
print(my_string) prints the object my_string as it is stored within R; that is, it prints enclosing quotes and literally prints special characters like \n rather than just inserting a new line
by contrast, writeLines(my_string) prints the object as an end-user would read it, excluding enclosing quotes and compiling special characters like \n so the end-user sees a new line inserted rather than seeing a literal \n
The writeLines() function:
?writeLines
# SYNTAX AND DEFAULT VALUES
writeLines(text, con = stdout(), sep = "\n", useBytes = FALSE)writeLines() displays quotes and backslashes as they would be read, rather than as R stores them.” (From writeLines documentation)
writeLines() to see how the escaped string lookswriteLines() will also output the string without showing the outer pair of double quotes that R uses to store it, so we only see the content of the stringtext: Character vector containing the text you want to display
The backslash (\) can also be used to form special characters, such as \n (newline character) and \t (tab character).
These characters following a backslash \ take on new meaning. For example, the n by itself is just a literal n. When you add a backslash to n, you are making it a special character where \n now represents a newline.
Use writeLines() to see the escaped string:
If
\is used as an escape character in regular expressions, how do you match a literal\? Well you need to escape it, creating the regular expression\\. To create that regular expression, you need to use a string, which also needs to escape\. That means to match a literal\you need to write\\\\— you need four backslashes to match one!
Credit: R for Data Science Strings Chapter
In regular expressions, the backslash \ also serves as an escape character, so in order to match a literal \, we need to use the regular expression \\. For example, let’s say we want to match the \ in the following string:
my_string <- "The executable is located in C:\\Program Files\\Git\\bin"
# Use writeLines() to see escaped string
writeLines(my_string)
#> The executable is located in C:\Program Files\Git\binThe regular expression we need is \\. But this doesn’t work:
Why is that? Let’s take a look at what is happening with the string "\\" we are providing as the pattern argument:
As seen, once escaped, the string "\\" becomes \ - so we were providing \ as the regular expression (i.e., pattern argument) instead of the \\ that we wanted. In order to get \\, we need to use the string "\\\\", where the 1st \ escapes the 2nd and the 3rd \ escapes the 4th:
# Use writeLines() to see the escaped string
writeLines("\\\\")
#> \\
# This properly matches the `\` in the string
str_view_all(string = my_string, pattern = "\\\\")
Summary: Whenever we need to use backslash in our regular expression, we’ll need to escape the backslash (by using another backslash) in the string that we provide as the regex pattern. For example, to match a newline character \n we need to use "\\n", to match a tab character \t we need to use "\\t", etc.
CRYSTAL - MAYBE HERE PROVIDE AN EXAMPLE OR IMAGE OF REGULAR EXPRESSIONS IN ACTION; GIVES A SENSE OF HOW THEY ARE USED BEFORE WE GET INTO MECHANICS OF THEM
Some common regular expression patterns include (not inclusive):
Credit: DaveChild Regular Expression Cheat Sheet
Select each tab
| STRING (type string that represents regex) |
REGEX (to have this appear in your regex) |
MATCHES (to match with this text) |
|---|---|---|
"\\d" |
\d |
any digit |
"\\D" |
\D |
any non-digit |
"\\s" |
\s |
any whitespace |
"\\S" |
\S |
any non-whitespace |
"\\w" |
\w |
any word character |
"\\W" |
\W |
any non-word character |
| Other regex involving backslashes… | ||
"\\n" |
\n |
newline |
"\\t" |
\t |
tab |
"\\\\" |
\\ |
\ |
"\\." |
\. |
. |
"\\?" |
\? |
? |
"\\(" |
\( |
( |
"\\)" |
\) |
) |
"\\{" |
\{ |
{ |
"\\}" |
\} |
} |
Credit: Working with strings in stringr Cheat sheet
There are certain character classes in regular expression that have special meaning. For example, \d is used to match any digit (i.e., number), \s is used to match any whitespace (i.e., space, tab, or newline character), and \w is used to match any word character (i.e., alphanumeric character or underscore).
“But wait… there’s more! Before a regex is interpreted as a regular expression, it is also interpreted by R as a string. And backslash is used to escape there as well. So, in the end, you need to preprend two backslashes…” This means in R, you would write out the regex patterns as "\\d", "\\s", "\\w", etc.
Credit: Escaping sequences from Stat 545
Example: Using
\d & \D to match digits & non-digits
We can use \d to match all instances of a digit (i.e., number):
# The escaped string "\\d" results in the regex \d
writeLines("\\d")
#> \d
# Match any instances of a digit
str_view_all(string = p12_df$text[119], pattern = "\\d")
We can use \D to match all instances of a non-digit character:
# The escaped string "\\D" results in the regex \D
writeLines("\\D")
#> \D
# Match any instances of a non-digit
str_view_all(string = p12_df$text[119], pattern = "\\D")
This matches all instances of a digit followed by a non-digit character:
Example: Using
\s & \S to match whitespace & non-whitespace
We can use \s to match all instances of a whitespace (i.e., space, tab, or newline character):
# The escaped string "\\s" results in the regex \s
writeLines("\\s")
#> \s
# Match any instances of a whitespace
str_view_all(string = p12_df$text[119], pattern = "\\s")
We can use \S to match all instances of a non-whitespace character:
# The escaped string "\\S" results in the regex \S
writeLines("\\S")
#> \S
# Match any instances of a non-whitespace
str_view_all(string = p12_df$text[119], pattern = "\\S")
This matches all instances of the letter e followed by a whitespace character:
Example: Using
\w & \W to match words & non-words
We can use \w to match all instances of a word character (i.e., alphanumeric character or underscore):
# The escaped string "\\w" results in the regex \w
writeLines("\\w")
#> \w
# Match any instances of a word character
str_view_all(string = p12_df$text[119], pattern = "\\w")
We can use \W to match all instances of a non-word character:
# The escaped string "\\W" results in the regex \W
writeLines("\\W")
#> \W
# Match any instances of a non-word character
str_view_all(string = p12_df$text[119], pattern = "\\W")
This matches all instances of 3-letter words:
The second half of the table above shows other regular expressions involving backslashes. This includes special characters like \n and \t, as well as using backslash to escape characters that have special meanings in regex, like . or ? (as we will soon see). So to match a literal period or question mark, we need to use the regex \. and \?, or strings "\\." and "\\?" in R.
| Character | Description |
|---|---|
* |
0 or more |
? |
0 or 1 |
+ |
1 or more |
{3} |
Exactly 3 |
{3,} |
3 or more |
{3,5} |
3, 4, or 5 |
We can use quantifiers to specify the amount of a certain character or expression to match. The quantifier should directly follow the pattern you want to quantify. For example, s? matches 0 or 1 s and \d{4} matches exactly 4 digits.
Example: Using the
*, ?, and + quantifiers
We can use * to match 0 or more of a pattern:
# Matches all instances of `s` followed by 0 or more non-word character
str_view_all(string = p12_df$text[119], pattern = "s\\W*")
We can use ? to match 0 or 1 of a pattern:
# Matches all instances of `s` followed by 0 or 1 non-word character
str_view_all(string = p12_df$text[119], pattern = "s\\W?")
We can use + to match 1 or more of a pattern:
Example: Using
{...} to specify how many occurrences to match
We can use {n} to specify the exact number of characters or expressions to match:
# Matches words with exactly 3 letters
str_view_all(string = p12_df$text[119], pattern = "\\s\\w{3}\\s")
We can use {n,} to specify n as the minimum amount to match:
# Matches words with 3 or more letters
str_view_all(string = p12_df$text[119], pattern = "\\s\\w{3,}\\s")
We can use {n,m} to specify we want to match between n and m amount (inclusive):
| String | Character | Description |
|---|---|---|
"^" |
^ |
Start of string, or start of line in multi-line pattern |
"$" |
$ |
End of string, or end of line in multi-line pattern |
"\\b" |
\b |
Word boundary |
"\\B" |
\B |
Non-word boundary |
We can use anchors to indicate which part of the string to match. For example, ^ matches the start of the string, $ matches the end of the string (Notice how we do not need to escape these characters). \b can be used to help detect word boundaries, and \B can be used to help match characters within a word.
Example: Using
^ & $ to match start & end of string
We can use ^ to match the start of a string:
# Matches only the quotation mark at the start of the text and not the end quote
str_view_all(string = p12_df$text[119], pattern = '^"')
We can use $ to match the end of a string:
Example: Using
\b & \B to match word boundary & non-word boundary
We can use \b to help detect word boundary:
# Matches words with 3 or more letters using \b
str_view_all(string = p12_df$text[119], pattern = "\\b\\w{3,}\\b")Notice how this is much flexible than trying to use whitespace (\s) to determine word boundary:
# Matches words with 3 or more letters using \s
str_view_all(string = p12_df$text[119], pattern = "\\s\\w{3,}\\s")
We can use \B to help match characters within a word:
| Character | Description |
|---|---|
. |
Match any character except newline (\n) |
a|b |
Match a or b |
[abc] |
Match either a, b, or c |
[^abc] |
Match anything except a, b, or c |
[a-z] |
Match range of lowercase letters from a to z |
[A-Z] |
Match range of uppercase letters from A to Z |
[0-9] |
Match range of numbers from 0 to 9 |
The table above lists some more ways regular expression offers us flexibility and option in what we want to match. The period . acts as a wildcard to match any character except newline. The vertical bar | is similar to an OR operator. Square brackets [...] can be used to specify a set or range of characters to match (or not to match).
Example: Using
. as a wildcard
We can use . to match any character except newline (\n):
We can confirm there is a newline in the tweet above by using writeLines():
Example: Using
| as an OR operator
We can use | to match either one of multiple patterns:
Example: Using
[...] to match (or not match) a set or range of characters
We can use [...] to match any set of characters:
# Matches any 2 consecutive vowels
str_view_all(string = p12_df$text[119], pattern = "[aeiouAEIOU]{2}")
We can also use [...] to match any range of alpha or numeric characters:
# Matches only lowercase x through z or uppercase A through C
str_view_all(string = p12_df$text[119], pattern = "[x-zA-C]")
# Matches only numbers 1 through 4 or the pound sign
str_view_all(string = p12_df$text[119], pattern = "[1-4#]")
We can use [^...] to indicate we do not want to match the provided set or range of characters:
# Matches anything that's not uppercase letters
str_view_all(string = p12_df$text[119], pattern = "[^A-Z]+")Notice that [...] only matches a single character (see second to last example above). We need to use quantifiers if we want to match a stretch of characters (see last example above).
| String | Character | Description |
|---|---|---|
"(...)" |
(...) |
Capturing group |
"(?:...)" |
(?:...) |
Non-capturing group |
"\\1" |
\1 |
Part of the string matched by capturing group 1 |
"\\2" |
\2 |
Part of the string matched by capturing group 2 |
| … | … | … |
Parentheses can be used to group parts of our regular expression together. Normal parentheses (...) creates what is called a numbered capturing group. “A capturing group stores the part of the string matched by the part of the regular expression inside the parentheses”. For example, if we have (\d), we can refer back to the digit matched by this capturing group using backreferences, like \1.
Credit: Hadley Wickham (R for Data Science) Grouping and backreferences
If we only want to use parentheses for grouping purposes and do not need to reference the matched values, we can use a non-capturing group (?:...).
Example: Using capturing groups
(...) and backreferences
We can use capturing groups (...) to match certain patterns, then reference what was matched:
Example: Using non-capturing groups
(?:...) for grouping purposes
We can use non-capturing groups (?:...) if we just want to group certain parts of the regex but don’t need to reference the matched value:
# Matches one or more of a digit followed by 3 letters
str_view_all(string = p12_df$text[119], pattern = "(?:\\d[A-Za-z]{3})+")
Normal parentheses (capturing groups) can still work for general grouping purposes too. But if you want to group things together without capturing them, you can just use non-capturing groups:
stringr functionsUsing regex in stringr functions (From R for Data Science)
When we specify a pattern in a stringr function, such as str_view(), it is automatically wrapped in a call to regex() (i.e., treated as a regular expression)
For simplicity, we can omit the call to regex()
But, there are additional arguments we can supply to regex() if we wanted
regex(pattern, ignore_case = FALSE, multiline = FALSE, comments = FALSE, ...)ignore_case: If TRUE, allows characters to match either their uppercase or lowercase formsmultiline: If TRUE, allows ^ and $ to match the start and end of each line rather than the start and end of the complete stringcomments: If TRUE, allows you to use comments and whitespace to make complex regular expressions more understandable
#"\\ "Example: Specifying
ignore_case = TRUE in regex()
Let’s say we have the following string:
We can match all the yay’s using the following regex:
Equivalently, we can specify ignore_case = TRUE to avoid dealing with casing variations:
str_detect()
The str_detect() function:
TRUE if there is a match, FALSE if there is not)string: Character vector (or vector coercible to character) to searchpattern: Pattern to look fornegate: If set to TRUE, the returned logical vector will contain TRUE if there is not a match and FALSE if there is oneExample: Using
str_detect() on string
Example: Using
str_detect() on character vector
Example: Using
str_detect() on dataframe column
Let’s create new columns in p12_df called is_am and is_pm that indicates whether or not each tweet’s created_at time is in the AM or PM, respectively:
p12_df %>%
mutate(
# Returns `TRUE` if the hour is 0#, 10, or 11, `FALSE` otherwise
is_am = str_detect(string = created_at, pattern = " 0\\d| 1[01]"),
# Recall we can set the `negate` argument to switch the returned `TRUE`/`FALSE`
is_pm = str_detect(string = created_at, pattern = " 0\\d| 1[01]", negate = TRUE)
) %>% select(created_at, is_am, is_pm)
#> # A tibble: 328 x 3
#> created_at is_am is_pm
#> <dttm> <lgl> <lgl>
#> 1 2020-04-25 22:37:18 FALSE TRUE
#> 2 2020-04-23 21:11:49 FALSE TRUE
#> 3 2020-04-21 04:00:00 TRUE FALSE
#> 4 2020-04-24 03:00:00 TRUE FALSE
#> 5 2020-04-20 19:00:21 FALSE TRUE
#> 6 2020-04-20 02:20:01 TRUE FALSE
#> 7 2020-04-22 04:00:00 TRUE FALSE
#> 8 2020-04-25 17:00:00 FALSE TRUE
#> 9 2020-04-21 15:13:06 FALSE TRUE
#> 10 2020-04-21 17:52:47 FALSE TRUE
#> # ... with 318 more rows
Because TRUE evaluates to 1 and FALSE evaluates to 0 in a numerical context, we could also sum the returned logical vector to see how many of the elements in the vector had a match:
# Number of tweets that were created in the AM
num_am_tweets <- sum(str_detect(string = p12_df$created_at, pattern = " 0\\d| 1[01]"))
num_am_tweets
#> [1] 53
Additionally, we can take the average of the logical vector to get the proportion of elements in the input vector that had a match:
# Proportion of tweets that were created in the AM
pct_am_tweets <- mean(str_detect(string = p12_df$created_at, pattern = " 0\\d| 1[01]"))
pct_am_tweets
#> [1] 0.1615854
We can also use the logical vector returned from str_detect() to filter p12_df to only include rows that had a match:
# Keep only rows whose tweet was created in the AM
p12_df %>%
filter(str_detect(string = created_at, pattern = " 0\\d| 1[01]"))
#> # A tibble: 53 x 5
#> user_id created_at screen_name text location
#> <chr> <dttm> <chr> <chr> <chr>
#> 1 22080148 2020-04-21 04:00:00 WSUPullman "Darien McLaughlin '19~ Pullman, W~
#> 2 22080148 2020-04-24 03:00:00 WSUPullman "6 houses, one pick. C~ Pullman, W~
#> 3 22080148 2020-04-20 02:20:01 WSUPullman "Tell us one of your B~ Pullman, W~
#> 4 22080148 2020-04-22 04:00:00 WSUPullman "We loved seeing your ~ Pullman, W~
#> 5 22080148 2020-04-24 01:58:04 WSUPullman "#WSU agricultural sci~ Pullman, W~
#> 6 22080148 2020-04-22 02:22:03 WSUPullman "Nice \U0001f44d https~ Pullman, W~
#> 7 15988549 2020-04-20 02:52:31 CalAdmissio~ "@PaulineARoxas Congra~ Berkeley, ~
#> 8 15988549 2020-04-22 03:07:00 CalAdmissio~ "It’s time to make thi~ Berkeley, ~
#> 9 15988549 2020-04-22 00:00:08 CalAdmissio~ "Are you a #BerkeleyBo~ Berkeley, ~
#> 10 15988549 2020-04-20 03:03:21 CalAdmissio~ "@N48260756 We suggest~ Berkeley, ~
#> # ... with 43 more rowsstr_subset()
The str_subset() function:
string: Character vector (or vector coercible to character) to searchpattern: Pattern to look fornegate: If set to TRUE, the returned vector will contain only elements that did not match the specified patternExample: Using
str_subset() on character vector
Example: Using
str_subset() on dataframe column
# Subsets the `created_at` vector of `p12_df` to only keep elements that occured in the AM
str_subset(string = p12_df$created_at, pattern = " 0\\d| 1[01]")
#> [1] "2020-04-21 04:00:00" "2020-04-24 03:00:00" "2020-04-20 02:20:01"
#> [4] "2020-04-22 04:00:00" "2020-04-24 01:58:04" "2020-04-22 02:22:03"
#> [7] "2020-04-20 02:52:31" "2020-04-22 03:07:00" "2020-04-22 00:00:08"
#> [10] "2020-04-20 03:03:21" "2020-04-22 00:47:00" "2020-04-23 06:34:00"
#> [13] "2020-04-23 04:06:49" "2020-04-19 03:32:21" "2020-04-20 02:53:38"
#> [16] "2020-04-20 02:53:14" "2020-04-20 03:04:11" "2020-04-19 03:30:14"
#> [19] "2020-04-20 02:58:55" "2020-04-19 05:37:00" "2020-04-21 02:34:00"
#> [22] "2020-04-20 00:15:07" "2020-04-25 04:18:29" "2020-04-25 00:00:01"
#> [25] "2020-04-21 02:33:00" "2020-04-24 01:00:01" "2020-04-23 02:38:46"
#> [28] "2020-04-24 04:48:28" "2020-04-24 01:06:33" "2020-04-25 04:48:08"
#> [31] "2020-04-22 00:10:43" "2020-04-21 05:58:12" "2020-04-24 01:41:19"
#> [34] "2020-04-24 01:42:44" "2020-04-24 01:43:11" "2020-04-23 02:45:24"
#> [37] "2020-04-20 00:44:42" "2020-04-24 01:41:13" "2020-04-25 00:26:02"
#> [40] "2020-04-25 00:31:23" "2020-04-25 00:46:40" "2020-04-25 00:20:36"
#> [43] "2020-04-20 00:09:58" "2020-04-20 00:09:46" "2020-04-20 00:10:08"
#> [46] "2020-04-25 00:29:12" "2020-04-22 01:45:02" "2020-04-23 02:00:14"
#> [49] "2020-04-25 00:34:47" "2020-04-24 02:11:51" "2020-04-25 00:05:59"
#> [52] "2020-04-21 04:14:11" "2020-04-23 02:13:21"str_extract() & str_extract_all()
The str_extract() & str_extract_all() functions:
?str_extract
?str_extract_all
# SYNTAX AND DEFAULT VALUES
str_extract(string, pattern)
str_extract_all(string, pattern, simplify = FALSE)str_extract()) or all matches (str_extract_all()) for input vectorstring: Character vector (or vector coercible to character) to searchpattern: Pattern to look forsimplify: If set to TRUE, the returned matches will be in a character matrix rather than the default list of character vectorsExample: Using
str_extract() & str_extract_all() on character vector
[str_extract()] Extract the first occurrence of a word for each string:
# Extracts first match of a word
str_extract(string = c("Three French hens", "Two turtle doves", "A partridge in a pear tree"),
pattern = "\\w+")
#> [1] "Three" "Two" "A"[str_extract_all()] Extract all occurrences of a word for each string:
# Extracts all matches of a word, returning a list of character vectors
str_extract_all(string = c("Three French hens", "Two turtle doves", "A partridge in a pear tree"),
pattern = "\\w+")
#> [[1]]
#> [1] "Three" "French" "hens"
#>
#> [[2]]
#> [1] "Two" "turtle" "doves"
#>
#> [[3]]
#> [1] "A" "partridge" "in" "a" "pear" "tree"
# Extracts all matches of a word, returning a character matrix
str_extract_all(string = c("Three French hens", "Two turtle doves", "A partridge in a pear tree"),
pattern = "\\w+", simplify = TRUE)
#> [,1] [,2] [,3] [,4] [,5] [,6]
#> [1,] "Three" "French" "hens" "" "" ""
#> [2,] "Two" "turtle" "doves" "" "" ""
#> [3,] "A" "partridge" "in" "a" "pear" "tree"Example: Using
str_extract() & str_extract_all() on dataframe column
[str_extract()] Extract first hashtag:
# Extracts first match of a hashtag (if there is one)
p12_df %>%
mutate(
hashtag = str_extract(string = text, pattern = "#\\S+")
) %>% select(text, hashtag)
#> # A tibble: 328 x 2
#> text hashtag
#> <chr> <chr>
#> 1 "Big Dez is headed to Indy!\n\n#GoCougs | #NFLDraft2020 | @dadpat7 ~ #GoCougs
#> 2 "Cougar Cheese. That's it. That's the tweet. \U0001f9c0#WSU #GoCoug~ #WSU
#> 3 "Darien McLaughlin '19, and her dog, Yuki, went on a #Pullman dista~ #Pullman
#> 4 "6 houses, one pick. Cougs, which one you got? Reply <U+2B07><U+FE0F> #WSU #Coug~ #WSU
#> 5 "Why did you choose to attend @WSUPullman?\U0001f914 #WSU #GoCougs ~ #WSU
#> 6 "Tell us one of your Bryan Clock Tower memories <U+23F0> \U0001f43e #WSU #~ #WSU
#> 7 "We loved seeing your top three @WSUPullman buildings, but what are~ #WSU
#> 8 "Congratulations, graduates! We’re two weeks away from the #WSU sys~ #WSU
#> 9 "Learn more about this story at https://t.co/45BzKc2rFE. #WSU #GoCo~ #WSU
#> 10 "Tomorrow, our @WSUEsports Team is facing off against \n@Esports_WA~ #GoCoug~
#> # ... with 318 more rows[str_extract_all()] Extract all hashtags:
# Extracts all matches of hashtags (if there are any)
p12_df %>%
mutate(
hashtag_vector = str_extract_all(string = text, pattern = "#\\S+"),
# Use `as.character()` so we can see the content of the character vector of matches
hashtags = as.character(hashtag_vector)
) %>% select(text, hashtag_vector, hashtags)
#> # A tibble: 328 x 3
#> text hashtag_vector hashtags
#> <chr> <list> <chr>
#> 1 "Big Dez is headed to Indy!\n\n#GoC~ <chr [3]> "c(\"#GoCougs\", \"#NFLD~
#> 2 "Cougar Cheese. That's it. That's t~ <chr [2]> "c(\"#WSU\", \"#GoCougs\~
#> 3 "Darien McLaughlin '19, and her dog~ <chr [3]> "c(\"#Pullman\", \"#Coug~
#> 4 "6 houses, one pick. Cougs, which o~ <chr [3]> "c(\"#WSU\", \"#CougsCon~
#> 5 "Why did you choose to attend @WSUP~ <chr [2]> "c(\"#WSU\", \"#GoCougs\~
#> 6 "Tell us one of your Bryan Clock To~ <chr [2]> "c(\"#WSU\", \"#GoCougs\~
#> 7 "We loved seeing your top three @WS~ <chr [2]> "c(\"#WSU\", \"#GoCougs\~
#> 8 "Congratulations, graduates! We’re ~ <chr [3]> "c(\"#WSU\", \"#CougGrad~
#> 9 "Learn more about this story at htt~ <chr [2]> "c(\"#WSU\", \"#GoCougs\~
#> 10 "Tomorrow, our @WSUEsports Team is ~ <chr [1]> "#GoCougs!"
#> # ... with 318 more rowsstr_match() & str_match_all()
The str_match() & str_match_all() functions:
string: Character vector (or vector coercible to character) to searchpattern: Pattern to look forExample: Using
str_match() & str_match_all() on character vector
[str_match()] Extract the first month, day, year for each string:
# Extracts first match of month, day, year
str_match(string = c("5-1-2020", "12/25/17", "01.01.13 to 01.01.14"),
pattern = "(\\d+)[-/\\.](\\d+)[-/\\.](\\d+)")
#> [,1] [,2] [,3] [,4]
#> [1,] "5-1-2020" "5" "1" "2020"
#> [2,] "12/25/17" "12" "25" "17"
#> [3,] "01.01.13" "01" "01" "13"[str_match_all()] Extract all month, day, year for each string:
# Extracts all matches of month, day, year
str_match_all(string = c("5-1-2020", "12/25/17", "01.01.13 to 01.01.14"),
pattern = "(\\d+)[-/\\.](\\d+)[-/\\.](\\d+)")
#> [[1]]
#> [,1] [,2] [,3] [,4]
#> [1,] "5-1-2020" "5" "1" "2020"
#>
#> [[2]]
#> [,1] [,2] [,3] [,4]
#> [1,] "12/25/17" "12" "25" "17"
#>
#> [[3]]
#> [,1] [,2] [,3] [,4]
#> [1,] "01.01.13" "01" "01" "13"
#> [2,] "01.01.14" "01" "01" "14"Example: Using
str_match() on dataframe column
Below, we extract datetime from the created_at column. The first capturing group matches the date part and the second capturing group matches the time part:
datetime_regex <- "([\\d-]+) ([\\d:]+)"
p12_df %>%
mutate(
# The 1st capturing group will be in the 2nd column of the matrix returned from `str_match()`
# So we use [, 2] below and save the result to the `date` column of the dataframe
date = str_match(string = created_at, pattern = datetime_regex)[, 2],
# The 2nd capturing group will be in the 3rd column of the matrix returned from `str_match()`
# So we use [, 3] below and save the result to the `time` column of the dataframe
time = str_match(string = created_at, pattern = datetime_regex)[, 3]
) %>% select(created_at, date, time)
#> # A tibble: 328 x 3
#> created_at date time
#> <dttm> <chr> <chr>
#> 1 2020-04-25 22:37:18 2020-04-25 22:37:18
#> 2 2020-04-23 21:11:49 2020-04-23 21:11:49
#> 3 2020-04-21 04:00:00 2020-04-21 04:00:00
#> 4 2020-04-24 03:00:00 2020-04-24 03:00:00
#> 5 2020-04-20 19:00:21 2020-04-20 19:00:21
#> 6 2020-04-20 02:20:01 2020-04-20 02:20:01
#> 7 2020-04-22 04:00:00 2020-04-22 04:00:00
#> 8 2020-04-25 17:00:00 2020-04-25 17:00:00
#> 9 2020-04-21 15:13:06 2020-04-21 15:13:06
#> 10 2020-04-21 17:52:47 2020-04-21 17:52:47
#> # ... with 318 more rowsstr_replace() & str_replace_all()
The str_replace() & str_replace_all() functions:
?str_replace
?str_replace_all
# SYNTAX
str_replace(string, pattern, replacement)
str_replace_all(string, pattern, replacement)str_replace()) or all matches (str_replace_all()) for each string replaced with specified replacementstring: Character vector (or vector coercible to character) to searchpattern: Pattern to look forreplacement: What the matched pattern should be replaced withstr_replace_all() also supports multiple replacements, where you can omit the replacement argument and just provide a named vector of replacements as the patternExample: Using
str_replace() & str_replace_all()
[str_replace()] Replace the first occurrence of a vowel:
# Replace first vowel with empty string
str_replace(string = "Thanks for the Memories", pattern = "[aeiou]", replacement = "")
#> [1] "Thnks for the Memories"[str_replace_all()] Replace all occurrences of a vowel:
Example: Using backreferences with
str_replace() & str_replace_all()
[str_replace()] Reorders the first date that is matched:
# Use \\1, \\2, and \\3 to refer to the capturing groups (ie. month, day, year)
str_replace(string = "12/31/19 to 01/01/20", pattern = "(\\d+)/(\\d+)/(\\d+)",
replacement = "20\\3-\\1-\\2")
#> [1] "2019-12-31 to 01/01/20"[str_replace_all()] Reorders all dates that are matched:
Example: Using
str_replace_all() for multiple replacements
Example: Using
str_replace_all() on dataframe column
p12_df %>%
mutate(
# Replace all hashtags and handles from tweet with an empty string
removed_hashtags_handles = str_replace_all(string = text, pattern = "[@#]\\S+", replacement = "")
) %>% select(text, removed_hashtags_handles)
#> # A tibble: 328 x 2
#> text removed_hashtags_handles
#> <chr> <chr>
#> 1 "Big Dez is headed to Indy!\n\n#Go~ "Big Dez is headed to Indy!\n\n | | | ~
#> 2 "Cougar Cheese. That's it. That's ~ "Cougar Cheese. That's it. That's the tw~
#> 3 "Darien McLaughlin '19, and her do~ "Darien McLaughlin '19, and her dog, Yuk~
#> 4 "6 houses, one pick. Cougs, which ~ "6 houses, one pick. Cougs, which one yo~
#> 5 "Why did you choose to attend @WSU~ "Why did you choose to attend https:/~
#> 6 "Tell us one of your Bryan Clock T~ "Tell us one of your Bryan Clock Tower m~
#> 7 "We loved seeing your top three @W~ "We loved seeing your top three buildin~
#> 8 "Congratulations, graduates! We’re~ "Congratulations, graduates! We’re two w~
#> 9 "Learn more about this story at ht~ "Learn more about this story at https://~
#> 10 "Tomorrow, our @WSUEsports Team is~ "Tomorrow, our Team is facing off again~
#> # ... with 318 more rowsstr_split()
The str_split() function:
string: Character vector (or vector coercible to character) to searchpattern: Pattern to look for and split byn: Maximum number of substrings to returnsimplify: If set to TRUE, the returned matches will be in a character matrix rather than the default list of character vectorsExample: Using
str_split() on character vector
# Split by comma or the word "and"
str_split(string = c("The Lion, the Witch, and the Wardrobe", "Peanut butter and jelly"),
pattern = ",? and |, ")
#> [[1]]
#> [1] "The Lion" "the Witch" "the Wardrobe"
#>
#> [[2]]
#> [1] "Peanut butter" "jelly"
We can specify n to control the maximum number of substrings we want to return:
# Limit split to only return 2 substrings
str_split(string = c("The Lion, the Witch, and the Wardrobe", "Peanut butter and jelly"),
pattern = ",? and |, ", n = 2)
#> [[1]]
#> [1] "The Lion" "the Witch, and the Wardrobe"
#>
#> [[2]]
#> [1] "Peanut butter" "jelly"
We can specify simplify = TRUE to return a character matrix instead of a list:
Example: Using
str_split() on dataframe column
When we split the created_at field at either a hyphen or space, we can separated out the year, month, day, and time components of the string:
p12_df %>%
mutate(
# Use `as.character()` so we can see the content of the character vector of splitted strings
year_month_day_time = as.character(str_split(string = created_at, pattern = "[- ]"))
) %>% select(created_at, year_month_day_time)
#> # A tibble: 328 x 2
#> created_at year_month_day_time
#> <dttm> <chr>
#> 1 2020-04-25 22:37:18 "c(\"2020\", \"04\", \"25\", \"22:37:18\")"
#> 2 2020-04-23 21:11:49 "c(\"2020\", \"04\", \"23\", \"21:11:49\")"
#> 3 2020-04-21 04:00:00 "c(\"2020\", \"04\", \"21\", \"04:00:00\")"
#> 4 2020-04-24 03:00:00 "c(\"2020\", \"04\", \"24\", \"03:00:00\")"
#> 5 2020-04-20 19:00:21 "c(\"2020\", \"04\", \"20\", \"19:00:21\")"
#> 6 2020-04-20 02:20:01 "c(\"2020\", \"04\", \"20\", \"02:20:01\")"
#> 7 2020-04-22 04:00:00 "c(\"2020\", \"04\", \"22\", \"04:00:00\")"
#> 8 2020-04-25 17:00:00 "c(\"2020\", \"04\", \"25\", \"17:00:00\")"
#> 9 2020-04-21 15:13:06 "c(\"2020\", \"04\", \"21\", \"15:13:06\")"
#> 10 2020-04-21 17:52:47 "c(\"2020\", \"04\", \"21\", \"17:52:47\")"
#> # ... with 318 more rowsstr_count()
The str_count() function:
string: Character vector (or vector coercible to character) to searchpattern: Pattern to look forExample: Using
str_count() on character vector
Example: Using
str_count() on dataframe column
p12_df %>%
mutate(
# Counts the total number of hashtags and mentions
num_hashtags_and_mentions = str_count(string = text, pattern = "[@#]\\S+")
) %>% select(text, num_hashtags_and_mentions)
#> # A tibble: 328 x 2
#> text num_hashtags_and_ment~
#> <chr> <int>
#> 1 "Big Dez is headed to Indy!\n\n#GoCougs | #NFLDraft20~ 5
#> 2 "Cougar Cheese. That's it. That's the tweet. \U0001f9~ 2
#> 3 "Darien McLaughlin '19, and her dog, Yuki, went on a ~ 4
#> 4 "6 houses, one pick. Cougs, which one you got? Reply ~ 3
#> 5 "Why did you choose to attend @WSUPullman?\U0001f914 ~ 3
#> 6 "Tell us one of your Bryan Clock Tower memories <U+23F0> \U0~ 2
#> 7 "We loved seeing your top three @WSUPullman buildings~ 3
#> 8 "Congratulations, graduates! We’re two weeks away fro~ 3
#> 9 "Learn more about this story at https://t.co/45BzKc2r~ 2
#> 10 "Tomorrow, our @WSUEsports Team is facing off against~ 5
#> # ... with 318 more rowsstr_locate() & str_locate_all()
The str_locate() & str_locate_all() functions:
string: Character vector (or vector coercible to character) to searchpattern: Pattern to look forExample: Using
str_locate() & str_locate_all() on character vector
[str_locate()] Locate the start and end positions for first stretch of numbers:
# Locate positions for first stretch of numbers
str_locate(string = c("555.123.4567", "(555) 135-7900 and (555) 246-8000"),
pattern = "\\d+")
#> start end
#> [1,] 1 3
#> [2,] 2 4[str_locate_all()] Locate the start and end positions for all stretches of numbers:
# Locate positions for all stretches of numbers
str_locate_all(string = c("555.123.4567", "(555) 135-7900 and (555) 246-8000"),
pattern = "\\d+")
#> [[1]]
#> start end
#> [1,] 1 3
#> [2,] 5 7
#> [3,] 9 12
#>
#> [[2]]
#> start end
#> [1,] 2 4
#> [2,] 7 9
#> [3,] 11 14
#> [4,] 21 23
#> [5,] 26 28
#> [6,] 30 33Example: Using
str_locate() on dataframe column
p12_df %>%
mutate(
# Start position of first hashtag in tweet (ie. 1st column of matrix returned from `str_locate()`)
start_of_first_hashtag = str_locate(string = text, pattern = "#\\S+")[, 1],
# End position of first hashtag in tweet (ie. 2nd column of matrix returned from `str_locate()`)
end_of_first_hashtag = str_locate(string = text, pattern = "#\\S+")[, 2],
# Length of first hashtag in tweet (ie. difference between start and end positions)
length_of_first_hashtag = end_of_first_hashtag - start_of_first_hashtag
) %>% select(text, start_of_first_hashtag, end_of_first_hashtag, length_of_first_hashtag)
#> # A tibble: 328 x 4
#> text start_of_first_ha~ end_of_first_has~ length_of_first_~
#> <chr> <int> <int> <int>
#> 1 "Big Dez is headed to~ 29 36 7
#> 2 "Cougar Cheese. That'~ 46 49 3
#> 3 "Darien McLaughlin '1~ 53 60 7
#> 4 "6 houses, one pick. ~ 57 60 3
#> 5 "Why did you choose t~ 44 47 3
#> 6 "Tell us one of your ~ 52 55 3
#> 7 "We loved seeing your~ 144 147 3
#> 8 "Congratulations, gra~ 59 62 3
#> 9 "Learn more about thi~ 57 60 3
#> 10 "Tomorrow, our @WSUEs~ 266 274 8
#> # ... with 318 more rowsRegular expressions are tricky. RegExplain makes it easier to see what you’re doing.
Credit: Garrick Aden-Buie (RegExplain)
RegExplain is an RStudio addin that allows the user to check their regex matching functions interactively.
Markup Language
“A markup language is a computer language that uses tags to define elements within a document. It is human-readable, meaning markup files contain standard words, rather than typical programming syntax.”
Credit: Markup Language from TechTerms
Hypertext Markup Language (HTML)
Intro to HTML (and CSS)
A Simple HTML Document (From w3schools)
<tagname> Content </tagname><!DOCTYPE html> to indicate it is an HTML document<html> element is the root element of an HTML page, where all other elements are nested<head> element contains meta information about the document (ie. not displayed on webpage)
<body> element contains the visible page content<!DOCTYPE html>
<html>
<head>
<title>Page Title</title>
</head>
<body>
<h1>My First Heading</h1>
<p>My first paragraph.</p>
</body>
</html>
What are attributes?
name="value"Credit: HTML attributes from W3schools
Some common attributes you may encounter:
The href attribute for an <a> tag (specifies url to link to):
<a href="https://www.w3schools.com">This is a link</a>The src attribute for an <img> tag (specifies image to display):
<img src="html_cheatsheet.jpg" />You can add more than one attribute to an element:
<img src="html_cheatsheet.jpg" width="200" height="300" />The class and id attributes are also commonly added to elements to be able to identify and select for them
classclass attribute can specify one or more class names for an HTML element. followed by the class name (more from GeekstoGeeks here)
HTML:
<div class="countries">
<h3>United States</h3>
<p class="place">Washington D.C.</p>
<img src="https://cdn.aarp.net/content/dam/aarp/travel/destination-guides/2018/03/1140-trv-dst-dc-main.imgcache.revd66f01d4a19adcecdb09fdacd4471fa8.jpg">
</div>
<div class="countries">
<h3>Mexico</h3>
<p class="place">Guadalajara</p>
<img src="https://cityofguadalajara.com/wp-content/uploads/2016/11/Centro-Historico-de-Guadalajara-800x288.jpg">
</div>CSS:
<style>
.countries {
background-color: #e6e6e6;
color: #336699;
margin: 10px;
padding: 15px;
}
.place {
color: black;
}
</style>Result:
Washington D.C
Guadalajara
Credit: HTML Classes from W3schools
idid attribute is used to specify one unique HTML element within the HTML document# followed by the id name (more from GeekstoGeeks here)
HTML:
CSS:
<style>
#banner {
background-color: #e6e6e6;
font-size: 40px;
padding: 20px;
text-align: center;
}
</style>Result:
Credit: HTML Id from W3schools
<!DOCTYPE html>
<html>
<head>
<title>Page title (in head tag)</title>
</head>
<body>
<h1>Title of level 1 heading</h1>
<p>My first paragraph.</p>
<p>My second paragraph.</p>
<p>Add some bold text <strong>right here</strong></p>
<p>Add some italics text <em>right here</em></p>
<p>Include a hyperlink tag within a paragraph tag. this book looks interesting : <a href="https://bookdown.org/rdpeng/rprogdatascience/">R Programming for Data Science</a></p>
<p>Include another hyperlink tag within a paragraph tag. chapter on <a href="https://bookdown.org/rdpeng/rprogdatascience/regular-expressions.html">Regular Expressions</a></p>
<p> put a button inside this paragraph <button>I am a button!</button></p>
<p>Here are some items in a list, but items not placed within an unordered list </p>
<li> text you want in item</li>
<li> text you want in another item</li>
<p>Here are some items in an unordered list</p>
<ul>
<li> first item in unordered list </li>
<li> second item in unordered list </li>
</ul>
</body>
</html>Lots of wonderful resources on the web to learn HTML!
rvestThe rvest package
rvesthelps you scrape information from web pages. It is designed to work with magrittr to make it easy to express common web scraping tasks, inspired by libraries like beautiful soup.
Credit: rvest webpage
[
rvestpackage contains] Wrappers around thexml2andhttrpackages to make it easy to download, then manipulate, HTML and XML.
Credit: rvest package documentation
Why use the rvest package?
rvest makes it easy to parse HTMLread_html() function to read in the HTML and convert it to an xml_document/xml_node objectxml_node object, we can easily traverse the nested nodes (ie. children elements) and parse the HTMLrvest comes with many helpful functions to search and extract various parts of the HTML
html_node()/html_nodes(): Search and extract node(s) (ie. HTML elements)html_text(): Extract the content between HTML tagshtml_attr()/html_attrs(): Extract the attribute(s) of HTML tagsThe read_html() function:
?read_html
# SYNTAX AND DEFAULT VALUES
read_html(x, encoding = "", ..., options = c("RECOVER", "NOERROR", "NOBLANKS"))x: The input can be a string containing HTML or url to the webpage you want to scrapervest xml_document/xml_node object and can be easily parsedas.character()Scraping HTML from a webpage:
View Page Source
read_html()Inspect (Chrome) or Inspect Element (Firefox) that will pop up a side panel
View Page Source (i.e., what is scraped), since it also reflects changes made to the HTML after the page was loaded (e.g., by JavaScript)Example: Using
read_html() to read in HTML from string
html <- read_html("<h1>This is a heading.</h1><p>This is a paragraph.</p>")
# View object
html
#> {html_document}
#> <html>
#> [1] <body>\n<h1>This is a heading.</h1>\n<p>This is a paragraph.</p>\n</body>
# View class of object
class(html)
#> [1] "xml_document" "xml_node"
# View raw HTML
as.character(html)
#> [1] "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\" \"http://www.w3.org/TR/REC-html40/loose.dtd\">\n<html><body>\n<h1>This is a heading.</h1>\n<p>This is a paragraph.</p>\n</body></html>\n"Example: Using
read_html() to scrape the page https://corona.help/
corona <- read_html("https://corona.help/")
# View object
corona
#> {html_document}
#> <html class="loading" lang="en" data-textdirection="ltr">
#> [1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
#> [2] <body class="horizontal-layout horizontal-menu dark-layout 2-columns navb ...
# View class of object
class(corona)
#> [1] "xml_document" "xml_node"The html_node() & html_nodes() functions:
x: An rvest xml_document/xml_node object (use read_html() to get this)css: Selector (can select by HTML tag name, its attributes, etc.)html_node() returns the first element that it finds as an rvest xml_node object
html_nodes() returns all elements that it finds as an rvest xml_nodeset object
as.character()
as.character(html_node(...))Selecting for HTML elements:
'p', 'table', etc..: '.my-class'#: '#my-id''table tr' (selects all rows within a table)Inspect (Chrome) or Inspect Element (Firefox) to bring up a side panelExample: Using
html_node() & html_nodes() to parse HTML string
Remember that the input to html_node()/html_nodes() should be an rvest xml_document/xml_node object, which we can obtain from read_html():
html <- read_html("<p>Paragraph #1</p><p>Paragraph #2</p><p>Paragraph #3</p>")
# View class of object
class(html)
#> [1] "xml_document" "xml_node"
# View raw HTML to see what elements are there
as.character(html)
#> [1] "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\" \"http://www.w3.org/TR/REC-html40/loose.dtd\">\n<html><body>\n<p>Paragraph #1</p>\n<p>Paragraph #2</p>\n<p>Paragraph #3</p>\n</body></html>\n"
If we search for the <p> element using html_node(), it will return the first result:
first_p <- html_node(html, 'p')
# View class of object
class(first_p)
#> [1] "xml_node"
# View raw HTML
as.character(first_p)
#> [1] "<p>Paragraph #1</p>\n"
If we search for the <p> element using html_nodes(), it will return all results:
all_p <- html_nodes(html, 'p')
# View class of object
class(all_p)
#> [1] "xml_nodeset"
# View raw HTML
as.character(all_p)
#> [1] "<p>Paragraph #1</p>\n" "<p>Paragraph #2</p>\n" "<p>Paragraph #3</p>"
Note that we could also use %>%:
Example: Using
html_node() & html_nodes() to parse https://corona.help/
Let’s revisit the HTML we scraped from https://corona.help/ in the previous example
View Page Source to check that the table element is indeed in the scraped HTML# Scraped HTML is stored in this `xml_document`/`xml_node` object
class(corona)
#> [1] "xml_document" "xml_node"
Select for the <table> element on that page using html_node():
# Since this table is the only table on the page, we can just use `html_node()`
corona_table <- corona %>% html_node('table')
corona_table
#> {html_node}
#> <table class="table table-striped table-hover-animation mb-0" id="table">
#> [1] <thead id="thead"><tr>\n<th>COUNTRY</th>\n <th>I ...
#> [2] <tbody>\n<tr>\n<td><a href="https://corona.help/country/united-states">\n ...
# View class of object
class(corona_table)
#> [1] "xml_node"
Select all rows in the table (i.e., <tr> elements) using html_nodes()
<tr> element (i.e., row) has <th>/<td> elements (i.e., column cells) nested within it, and not the other way aroundtable tr td:nth-child(1) selects the first cell in each row a.k.a. the first column in table)# We can chain `html_node()`/`html_nodes()` functions
corona_rows <- corona %>% html_node('table') %>% html_nodes('tr')
# Alternatively, we can use `table tr` as the selector to select all `tr` elements within a `table`
corona_rows <- corona %>% html_nodes('table tr')
# Investigate object
head(corona_rows) # View first few rows
#> {xml_nodeset (6)}
#> [1] <tr>\n<th>COUNTRY</th>\n <th>INFECTED</th>\n ...
#> [2] <tr>\n<td><a href="https://corona.help/country/united-states">\n ...
#> [3] <tr>\n<td><a href="https://corona.help/country/india">\n ...
#> [4] <tr>\n<td><a href="https://corona.help/country/brazil">\n ...
#> [5] <tr>\n<td><a href="https://corona.help/country/russia">\n ...
#> [6] <tr>\n<td><a href="https://corona.help/country/united-kingdom">\n ...
typeof(corona_rows)
#> [1] "list"
class(corona_rows)
#> [1] "xml_nodeset"
length(corona_rows) # Number of elements
#> [1] 227The following examples use the Coronavirus data from https://corona.help/
<td> elements from each row)View corona_rows we selected from previous example:
# View first few rows
head(corona_rows)
#> {xml_nodeset (6)}
#> [1] <tr>\n<th>COUNTRY</th>\n <th>INFECTED</th>\n ...
#> [2] <tr>\n<td><a href="https://corona.help/country/united-states">\n ...
#> [3] <tr>\n<td><a href="https://corona.help/country/india">\n ...
#> [4] <tr>\n<td><a href="https://corona.help/country/brazil">\n ...
#> [5] <tr>\n<td><a href="https://corona.help/country/russia">\n ...
#> [6] <tr>\n<td><a href="https://corona.help/country/united-kingdom">\n ...
corona_rows[1:5] # first five rows
#> {xml_nodeset (5)}
#> [1] <tr>\n<th>COUNTRY</th>\n <th>INFECTED</th>\n ...
#> [2] <tr>\n<td><a href="https://corona.help/country/united-states">\n ...
#> [3] <tr>\n<td><a href="https://corona.help/country/india">\n ...
#> [4] <tr>\n<td><a href="https://corona.help/country/brazil">\n ...
#> [5] <tr>\n<td><a href="https://corona.help/country/russia">\n ...
corona_rows[c(1)] # header row
#> {xml_nodeset (1)}
#> [1] <tr>\n<th>COUNTRY</th>\n <th>INFECTED</th>\n ...
corona_rows[1] # header row
#> {xml_nodeset (1)}
#> [1] <tr>\n<th>COUNTRY</th>\n <th>INFECTED</th>\n ...
Let’s convert this to raw HTML using as.character() to practice writing regular expressions. Refer back to this output to help you determine what pattern you want to match:
# Convert rows to raw HTML
rows <- as.character(corona_rows)[-c(1)] # [-c(1)] means skip header row
# View first few rows as raw HTML
writeLines(head(rows, 2)) # printing via writeLines() is much prettier than printing via print()
#> <tr>
#> <td><a href="https://corona.help/country/united-states">
#> <div style="height:100%;width:100%">United States</div>
#> </a></td>
#> <td class="text-warning">29,518,078</td>
#> <td class="text-warning text-bold-700">67,116</td>
#> <td class="text-danger">533,386</td>
#> <td class="text-danger text-bold-700">1,927</td>
#> <td class="text-success">20,074,687</td>
#> <td class="text-success text-bold-700">66,957</td>
#> <td class="text-warning">8,910,005</td>
#> <td class="text-danger">14,281</td>
#> <td class="text-warning">364,820,765</td>
#>
#>
#> </tr>
#>
#> <tr>
#> <td><a href="https://corona.help/country/india">
#> <div style="height:100%;width:100%">India</div>
#> </a></td>
#> <td class="text-warning">11,173,567</td>
#> <td class="text-warning text-bold-700">16,819</td>
#> <td class="text-danger">157,584</td>
#> <td class="text-danger text-bold-700">113</td>
#> <td class="text-success">10,838,018</td>
#> <td class="text-success text-bold-700">13,785</td>
#> <td class="text-warning">177,965</td>
#> <td class="text-danger">8,944</td>
#> <td class="text-warning">219,178,908</td>
#>
#>
#> </tr>
# Investgate object named `rows`, which is a character vector
typeof(rows)
#> [1] "character"
class(rows)
#> [1] "character"
length(rows)
#> [1] 226Example: Using
str_subset() to subset rows
Let’s filter for rows whose country name starts with 'United'. First, preview what our regular expression matches using str_view():
Inspect the output from str_detect(), which returns TRUE if there is a match and FALSE if not. For example, we see there is a TRUE for the first element (United States) and fifth element (United Kingdom):
Finally, subset rows by country name using str_subset(), which keeps elements of character vector for which str_detect() is TRUE (i.e., keeps elements where the pattern “matches”):
subset_by_country <- str_subset(string = rows, pattern = 'United \\w+')
writeLines(subset_by_country)
#> <tr>
#> <td><a href="https://corona.help/country/united-states">
#> <div style="height:100%;width:100%">United States</div>
#> </a></td>
#> <td class="text-warning">29,518,078</td>
#> <td class="text-warning text-bold-700">67,116</td>
#> <td class="text-danger">533,386</td>
#> <td class="text-danger text-bold-700">1,927</td>
#> <td class="text-success">20,074,687</td>
#> <td class="text-success text-bold-700">66,957</td>
#> <td class="text-warning">8,910,005</td>
#> <td class="text-danger">14,281</td>
#> <td class="text-warning">364,820,765</td>
#>
#>
#> </tr>
#>
#> <tr>
#> <td><a href="https://corona.help/country/united-kingdom">
#> <div style="height:100%;width:100%">United Kingdom</div>
#> </a></td>
#> <td class="text-warning">4,213,733</td>
#> <td class="text-warning text-bold-700">6,614</td>
#> <td class="text-danger">124,259</td>
#> <td class="text-danger text-bold-700">242</td>
#> <td class="text-success">3,096,564</td>
#> <td class="text-success text-bold-700">90,844</td>
#> <td class="text-warning">992,910</td>
#> <td class="text-danger">1,647</td>
#> <td class="text-warning">93,148,314</td>
#>
#>
#> </tr>
#>
#> <tr>
#> <td><a href="https://corona.help/country/united-arab-emirates">
#> <div style="height:100%;width:100%">United Arab Emirates</div>
#> </a></td>
#> <td class="text-warning">402,205</td>
#> <td class="text-warning text-bold-700">2,742</td>
#> <td class="text-danger">1,286</td>
#> <td class="text-danger text-bold-700">17</td>
#> <td class="text-success">387,278</td>
#> <td class="text-success text-bold-700">1,691</td>
#> <td class="text-warning">13,641</td>
#> <td class="text-danger">0</td>
#> <td class="text-warning">31,520,191</td>
#>
#>
#> </tr>Example: Using
str_extract() to extract link for each row
Since all links follow the same pattern, we can use regex to extract this info:
links <- str_extract(string = rows, pattern = 'https://corona.help/country/[-a-z]+')
# View first few links
head(links)
#> [1] "https://corona.help/country/united-states"
#> [2] "https://corona.help/country/india"
#> [3] "https://corona.help/country/brazil"
#> [4] "https://corona.help/country/russia"
#> [5] "https://corona.help/country/united-kingdom"
#> [6] "https://corona.help/country/france"Example: Using
str_match() to extract country for each row
Since all countries are in a div element with the same attributes, we can use the following regex to extract the country name:
countries <- str_match(string = rows, pattern = '<div style="height:100%;width:100%">([\\w ]+)</div>')
# View first few countries
# We used a capturing group to extract the country name from between the tags
head(countries)
#> [,1]
#> [1,] "<div style=\"height:100%;width:100%\">United States</div>"
#> [2,] "<div style=\"height:100%;width:100%\">India</div>"
#> [3,] "<div style=\"height:100%;width:100%\">Brazil</div>"
#> [4,] "<div style=\"height:100%;width:100%\">Russia</div>"
#> [5,] "<div style=\"height:100%;width:100%\">United Kingdom</div>"
#> [6,] "<div style=\"height:100%;width:100%\">France</div>"
#> [,2]
#> [1,] "United States"
#> [2,] "India"
#> [3,] "Brazil"
#> [4,] "Russia"
#> [5,] "United Kingdom"
#> [6,] "France"Example: Using
str_match_all() to extract number deaths and critical for each row
Since both the number of deaths and critical are in a <td> element with the same class attribute, we can use the following regex to extract both numbers:
num_danger <- str_match_all(string = rows, pattern = '<td class="text-danger">([\\d,]+)</td>')
# View matches for first few rows
# We used a capturing group to extract the numbers from between the tags
head(num_danger)
#> [[1]]
#> [,1] [,2]
#> [1,] "<td class=\"text-danger\">533,386</td>" "533,386"
#> [2,] "<td class=\"text-danger\">14,281</td>" "14,281"
#>
#> [[2]]
#> [,1] [,2]
#> [1,] "<td class=\"text-danger\">157,584</td>" "157,584"
#> [2,] "<td class=\"text-danger\">8,944</td>" "8,944"
#>
#> [[3]]
#> [,1] [,2]
#> [1,] "<td class=\"text-danger\">261,188</td>" "261,188"
#> [2,] "<td class=\"text-danger\">8,318</td>" "8,318"
#>
#> [[4]]
#> [,1] [,2]
#> [1,] "<td class=\"text-danger\">87,823</td>" "87,823"
#> [2,] "<td class=\"text-danger\">2,300</td>" "2,300"
#>
#> [[5]]
#> [,1] [,2]
#> [1,] "<td class=\"text-danger\">124,259</td>" "124,259"
#> [2,] "<td class=\"text-danger\">1,647</td>" "1,647"
#>
#> [[6]]
#> [,1] [,2]
#> [1,] "<td class=\"text-danger\">87,835</td>" "87,835"
#> [2,] "<td class=\"text-danger\">3,633</td>" "3,633"Example: Using
str_replace_all() to convert numeric values to thousands for each row
Rewrite all numeric values greater than one thousand in terms of k:
num_to_k <- str_replace_all(string = rows, pattern = '>([\\d,]+),\\d{3}<', replacement = '>\\1k<')
# View replacements for first few rows
writeLines(head(num_to_k))
#> <tr>
#> <td><a href="https://corona.help/country/united-states">
#> <div style="height:100%;width:100%">United States</div>
#> </a></td>
#> <td class="text-warning">29,518k</td>
#> <td class="text-warning text-bold-700">67k</td>
#> <td class="text-danger">533k</td>
#> <td class="text-danger text-bold-700">1k</td>
#> <td class="text-success">20,074k</td>
#> <td class="text-success text-bold-700">66k</td>
#> <td class="text-warning">8,910k</td>
#> <td class="text-danger">14k</td>
#> <td class="text-warning">364,820k</td>
#>
#>
#> </tr>
#>
#> <tr>
#> <td><a href="https://corona.help/country/india">
#> <div style="height:100%;width:100%">India</div>
#> </a></td>
#> <td class="text-warning">11,173k</td>
#> <td class="text-warning text-bold-700">16k</td>
#> <td class="text-danger">157k</td>
#> <td class="text-danger text-bold-700">113</td>
#> <td class="text-success">10,838k</td>
#> <td class="text-success text-bold-700">13k</td>
#> <td class="text-warning">177k</td>
#> <td class="text-danger">8k</td>
#> <td class="text-warning">219,178k</td>
#>
#>
#> </tr>
#>
#> <tr>
#> <td><a href="https://corona.help/country/brazil">
#> <div style="height:100%;width:100%">Brazil</div>
#> </a></td>
#> <td class="text-warning">10,796k</td>
#> <td class="text-warning text-bold-700">74k</td>
#> <td class="text-danger">261k</td>
#> <td class="text-danger text-bold-700">1k</td>
#> <td class="text-success">9,637k</td>
#> <td class="text-success text-bold-700">45k</td>
#> <td class="text-warning">898k</td>
#> <td class="text-danger">8k</td>
#> <td class="text-warning">28,600k</td>
#>
#>
#> </tr>
#>
#> <tr>
#> <td><a href="https://corona.help/country/russia">
#> <div style="height:100%;width:100%">Russia</div>
#> </a></td>
#> <td class="text-warning">4,290k</td>
#> <td class="text-warning text-bold-700">11k</td>
#> <td class="text-danger">87k</td>
#> <td class="text-danger text-bold-700">475</td>
#> <td class="text-success">3,869k</td>
#> <td class="text-success text-bold-700">16k</td>
#> <td class="text-warning">332k</td>
#> <td class="text-danger">2k</td>
#> <td class="text-warning">112,100k</td>
#>
#>
#> </tr>
#>
#> <tr>
#> <td><a href="https://corona.help/country/united-kingdom">
#> <div style="height:100%;width:100%">United Kingdom</div>
#> </a></td>
#> <td class="text-warning">4,213k</td>
#> <td class="text-warning text-bold-700">6k</td>
#> <td class="text-danger">124k</td>
#> <td class="text-danger text-bold-700">242</td>
#> <td class="text-success">3,096k</td>
#> <td class="text-success text-bold-700">90k</td>
#> <td class="text-warning">992k</td>
#> <td class="text-danger">1k</td>
#> <td class="text-warning">93,148k</td>
#>
#>
#> </tr>
#>
#> <tr>
#> <td><a href="https://corona.help/country/france">
#> <div style="height:100%;width:100%">France</div>
#> </a></td>
#> <td class="text-warning">3,835k</td>
#> <td class="text-warning text-bold-700">25k</td>
#> <td class="text-danger">87k</td>
#> <td class="text-danger text-bold-700">293</td>
#> <td class="text-success">262k</td>
#> <td class="text-success text-bold-700">1k</td>
#> <td class="text-warning">3,485k</td>
#> <td class="text-danger">3k</td>
#> <td class="text-warning">53,959k</td>
#>
#>
#> </tr>